Pre-trained language models achieve superior performance, but they are computationally expensive due to their large size. Techniques such as pruning and knowledge distillation (KD) have been developed to reduce their size and latency. In most structural pruning methods, the pruning units, such as attention heads and feed-forward hidden dimensions, only span a small model structure space and limit the structures that the pruning algorithm can explore. In this work, we propose Gradient-based Intra-attention pruning (GRAIN), which inspects fine intra-attention structures, and allows different heads to have different sizes. Intra-attention pruning greatly expands the searching space of model structures and yields highly heterogeneous structures. We further propose structure regularization to encourage generating more regular structures, which achieves higher speedups than heterogeneous ones. We also integrate KD into the pruning process with a gradient separation strategy to reduce the interference of KD with the pruning process. GRAIN is evaluated on a variety of tasks. Results show that it notably outperforms other methods at the same or similar model size. Even under extreme compression where only $3\%$ weights in transformers remain, the pruned model is still competitive.

Pre-trained Language Model (PLM) has become a representative foundation model in the natural language processing field. Most PLMs are trained with linguistic-agnostic pre-training tasks on the surface form of the text, such as the masked language model (MLM). To further empower the PLMs with richer linguistic features, in this paper, we aim to propose a simple but effective way to learn linguistic features for pre-trained language models. We propose LERT, a pre-trained language model that is trained on three types of linguistic features along with the original MLM pre-training task, using a linguistically-informed pre-training (LIP) strategy. We carried out extensive experiments on ten Chinese NLU tasks, and the experimental results show that LERT could bring significant improvements over various comparable baselines. Furthermore, we also conduct analytical experiments in various linguistic aspects, and the results prove that the design of LERT is valid and effective. Resources are available at https://github.com/ymcui/LERT

深度学习（DL）的快速增长和部署目睹了新兴的隐私和安全问题。为了减轻这些问题，已经讨论了安全的多方计算（MPC），以实现隐私保护DL计算。在实践中，它们通常是在很高的计算和沟通开销中，并有可能禁止其在大规模系统中的受欢迎程度。两种正交研究趋势吸引了人们对安全深度学习的能源效率的巨大兴趣，即MPC比较方案的高架降低和硬件加速度。但是，他们要么达到较低的减少比率，因此由于计算和通信节省有限而遭受了高潜伏期，或者是渴望的，因为现有的作品主要集中在CPU和GPU等一般计算平台上。在这项工作中，作为第一次尝试，我们通过将加密构件构建块的硬件延迟整合到DNN损耗功能中，以实现高能量效率，开发了一个系统的polympcnet，以减少MPC比较协议和硬件加速的联合额外降低的系统框架Polympcnet。和安全保证。我们的关键设计原理不是在DNN进行良好训练之后（通过删除或删除某些非物质操作员）训练（通过删除或删除某些非物质操作员）之后检查模型敏感性，而是要准确地执行DNN设计中的假设 - 培训DNN既是DNN都硬件有效且安全，同时逃脱了当地的最小值和鞍点并保持高精度。更具体地说，我们提出了通过多项式激活初始化方法直接提出的加密硬件友好的可训练多项式激活功能，以替代昂贵的2P-RELU操作员。我们开发了一个密码硬件调度程序和现场可编程门阵列（FPGA）平台的相应性能模型。

在本文中，我们介绍了CTC 2021的概述，这是针对母语人士的中文文本校正任务。我们详细描述了任务定义以及培训和评估的数据。我们还总结了该任务参与者调查的方法。我们希望为此任务收集和注释的数据集可以促进并加快该研究领域的未来发展。因此，伪培训数据，金标准验证数据和整个排行榜可在https://destwang.github.io/ctc2021-explorer/上在线公开获取。

本文旨在通过介绍第一个中国数学预训练的语言模型〜（PLM）来提高机器的数学智能，以有效理解和表示数学问题。与其他标准NLP任务不同，数学文本很难理解，因为它们在问题陈述中涉及数学术语，符号和公式。通常，它需要复杂的数学逻辑和背景知识来解决数学问题。考虑到数学文本的复杂性质，我们设计了一种新的课程预培训方法，用于改善由基本和高级课程组成的数学PLM的学习。特别是，我们首先根据位置偏见的掩盖策略执行令牌级预训练，然后设计基于逻辑的预训练任务，旨在分别恢复改组的句子和公式。最后，我们介绍了一项更加困难的预训练任务，该任务强制执行PLM以检测和纠正其生成的解决方案中的错误。我们对离线评估（包括九个与数学相关的任务）和在线$ A/B $测试进行了广泛的实验。实验结果证明了与许多竞争基线相比，我们的方法的有效性。我们的代码可在：\ textColor {blue} {\ url {https://github.com/rucaibox/jiuzhang}}}中获得。

Knowledge graph embedding (KGE) models learn the representation of entities and relations in knowledge graphs. Distance-based methods show promising performance on link prediction task, which predicts the result by the distance between two entity representations. However, most of these methods represent the head entity and tail entity separately, which limits the model capacity. We propose two novel distance-based methods named InterHT and InterHT+ that allow the head and tail entities to interact better and get better entity representation. Experimental results show that our proposed method achieves the best results on ogbl-wikikg2 dataset.

反复解决参数化最佳质量传输（POMT）问题是在图像配准和自适应网格生成的应用中的频繁任务。因此，开发一种高效的减少的解算器是至关重要的，该求解器同样准确为完整的订单模型。在本文中，我们通过调整用于非线性方程的专门设计的新的降低基础（RB）技术，提出了这种机器学习方法，该方法专门针对非线性方程设计，降低的剩余减少的过度搭配（R2-ROC）方法，对参数化的Monge- AMP $ \ agag {\ rm e} $重新等式。它构建在狭窄的模板有限不同方法（FDM）的顶部，是一个所谓的真理求解器，我们在本文中提出了与传输边界的Monge-AMP $ \ agr \ gamet {\ rm e} $重新等式。与R2-ROC方法一起，它使我们能够处理与Monge-AMP $ \ agom {\ RM e} $ RE方程有关的强大和独特的非线性，而不是诉诸非线性的任何直接近似的在线效率。有几个具有挑战性的数值测试展示了我们用各种参数边界条件解决Monge-AMP $ \ aga {\ rm e} $重新等式的方法的准确性和高效率。

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed implicitly, by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. CMT obtains 73.0% NDS on nuScenes benchmark. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code will be released at https://github.com/junjie18/CMT.

Knowledge graphs (KG) have served as the key component of various natural language processing applications. Commonsense knowledge graphs (CKG) are a special type of KG, where entities and relations are composed of free-form text. However, previous works in KG completion and CKG completion suffer from long-tail relations and newly-added relations which do not have many know triples for training. In light of this, few-shot KG completion (FKGC), which requires the strengths of graph representation learning and few-shot learning, has been proposed to challenge the problem of limited annotated data. In this paper, we comprehensively survey previous attempts on such tasks in the form of a series of methods and applications. Specifically, we first introduce FKGC challenges, commonly used KGs, and CKGs. Then we systematically categorize and summarize existing works in terms of the type of KGs and the methods. Finally, we present applications of FKGC models on prediction tasks in different areas and share our thoughts on future research directions of FKGC.

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited several support examples. In this work, we explore a simple yet unified solution for FSIS as well as its incremental variants, and introduce a new framework named Reference Twice (RefT) to fully explore the relationship between support/query features based on a Transformer-like framework. Our key insights are two folds: Firstly, with the aid of support masks, we can generate dynamic class centers more appropriately to re-weight query features. Secondly, we find that support object queries have already encoded key factors after base training. In this way, the query features can be enhanced twice from two aspects, i.e., feature-level and instance-level. In particular, we firstly design a mask-based dynamic weighting module to enhance support features and then propose to link object queries for better calibration via cross-attention. After the above steps, the novel classes can be improved significantly over our strong baseline. Additionally, our new framework can be easily extended to incremental FSIS with minor modification. When benchmarking results on the COCO dataset for FSIS, gFSIS, and iFSIS settings, our method achieves a competitive performance compared to existing approaches across different shots, e.g., we boost nAP by noticeable +8.2/+9.4 over the current state-of-the-art FSIS method for 10/30-shot. We further demonstrate the superiority of our approach on Few Shot Object Detection. Code and model will be available.